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» The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply - ' i 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30);DAYS. 

WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMU^^^ =: 1:.: , 

- Extensions Of tim6;nnay ibe; available linder the provisions of 37 CFR 1 . 1 36(a). In no event however, rnay a reply be timely fil^ • ' ' ' ; I ' - / 
after SIX (6) l\IlbNTlHSfrocn.th^hiailirjg date of this c^^ j 1 *• * . 

- If NO period for reply is specified abpye; the maxinrium statutory period will apply and wfll expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within (he set or extended period for reply will, by statute, cause the application to becorrie ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Offi(^ later than three months after the mailing date of this CGtmrriunication. even if timely filed, may reduce any« 
eamed patent term adjustment See 37 CFR 1.704(b). 

Status J ' 

1)^ Responsive to C(immunication(s) filetj on 13 July 2007 . ] 
2a)KI This action is FINAL I 2b)n This action Is non-final. ] 

3) D Since; this- appiica^^ in condition for allowance except for formal matters, prosecution as to th.e merits is 

close<j in ^ccdtjance^ ^rth the practice under Ex parte Qt^ay/e; 1 935 CD. 1 1 . 453 O.b. 213. 

Disposition of Claiims ; 

4) ^ Claimfs) 7.8.10-17.20.21.24 and 25 is/are pending in the aPDlication. ' . , 

4a) Of the above Glaim(s) |_ ; is/are withdra\vn from consideration. 

5) n Claim(s) is/are allowed. 

6) 13 Claim(s) 7.8. 10-m20j21.24 and 25 is/are rejected. |; J : 
?)□ Claim(s):LL_^ i^/^re biDjected to .| 

8) 0 Claim(s) . are subject to restriction and/or election requirement. 

Application Papers 

9) \3 The specification fe?objectfed to by the Examiner. 

10)K The drawing(s) filed on 27 August 2003 is/are: a)[SI accepted or b)n objected to by the Examiner. . 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 GFR 1 ,85(a).. .. 

Replacement drjawiiig sheet(s) including the correction is required if the drawing(s) Is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaratiori is objected to by the Examiner: Nbte the attached Office Action or fomri PTO-152. 

Priority under 35 U.S.C. § 119 

12)0 Acknowledgment is made,of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (fj. 

a)nAII b)n Some •* c)P None of: ' ; 

1 .□ Certified copies ^of the priority documents have been received. 

2.II] I Gertified Ggpiesjof the priority documents have been received in Application No: 'J \ 



3.D Copies of the ciertified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
See the attached detailed Office action for a list of the certified copies not received. . 
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11 1 ' detaiiLed action 



This aptibii is tesporisive to papers 'filed on July 13, 2007, ! 

j Response to Ameridment : '! 

2. 1^ ifespd^^ applicant's amendment received: on July 13, 2007, all reqjiested ehahges to 
the claims :hav€ been entered. Claims 24 and 25 hdve been added* Claims 1-6, 9, 18-19 and 22- 
23 have be^n cancelled. ! 



j :; i'jjjK i j| Response to Arguments \ 

3. AjppliGaht's arguments with respect to clainis 7, 16 and 20 have been considered but are 
moot in view of thje nevv ground(s) of rejection. 



!• I i 



Claim Rejections - SS JJSC §103 



The fGllowing is a quotation of 35 U.S.C. 103(a) which forms the basis for all 



obviousness rbjejctions spt forth in this Office action: ; , 

(a) A patent nfiay not be obtained though the invention is not identically disclosed or described as set forth in 
septipn 102 cif this title, if the differences between the subject matter sought to be patented and the prior, art are 
;/ supli ithat tih0 isiibjedt matter as a whole would have been obvious at the time the invention \yas made'to a person 
' having ordinary skfll !iri the art to which said subject matter !])ertains. Patentability shall not! be negatived by the 
nriannerin which th'e invention was made. : 



5. Claims 7, ISJ 10^13 and 15 are rejected' under 35 U.S.C. 103(a) as being linpateritable over 
Basu,;;alr,e4^yjof r^co in yievv of the article Embedded Bayesian Networks for Face I: ; • , 
Recognition to Netiari,' already of record. 



'I ■ • ■» ■ 
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As to claim ^7, Basu discloses a method comprising: 

V :! : ■! IM ! . ; ;, ■ : • i ■ i: : \ 

: R^cpgnizirig a face of a subject from first entries in a database (column 6, lines 32-58, 

wherein fhefdc^^ of an individual speaking is performed); ' 

Recogtii^ng audio-visual speech of the subject frpm second entries in the datable 
(column J; lines 18-42 and column II, lines 10-31, wherein speech is recognized using the audio 
of a audib'ryisual Signal and that speech is verified using the utterance verifier, which uses the 
video of the dtidMvisual signal); and • 

Identifying the subject based on recognizing the face and recognizing the audio-visual 
speech (column 8,^Jines 43-50, wherein the recognition results determined by the face andaudio- 
visual speech recognizer^^^ combined to produce a joint score which is used tp identify the 
individudrj: i . 

Basil dods.npt dischlose expressly including modeling an image including the face using 
an embedded hid|den Markov model (EHMM), wherein the EHMM is a hierarchical statistical 
model having a! paff^nti^^^ corresponding to a super i^ate of the EHMM and including a : 
plurality!of;nodes it6 represent hidden nodes, each node referring to a plurality of second nodes of 
a child layer corre^pohdiiig to a state of the EHMM, the plurality of second nodes each referring 
to an observation node; and wherein the state of the EHMM is described by a mixture of a 
pliiraiity of Gaussian density functions having diagonal covariance matrices. ; 

However, Nefian discloses recognizing a face of a subject from first entries in a database 
(Abstract), incluiJiWg modeling an image including the face using an embedded hidden ;Markov 
model (EHMM), wherein the E.HMM is a hierarchical (Statistical model having a parent layer 
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corresponding to a super state of the EHMM and including a plurality of nodes to represjent 
hidden nodes, eachinode referring to a plurality of second nodes of a child layer, corresponding to 
a state ojf the EHMIyl, the plurality of second nodes each referring to an observaition node and 
wherein the state of the EHMM is described by a nlixture of a plurality of Gaussian density 
functions hkving diagonal covariance matrices (Fig, 2a, equation 1 and Section 3, wherein 
square nodes correspond to the parent node and the circle nodes correspond to the child nodes 
or observation nodes dnd equation 1 corresponds to the mixture of Gaussian density furiciipns 
having diagoHahppvariance matrices), : 

BaSu & Nefian are combinable because they are from the same art of imjage.prdcessing, 
specifically recognition. : 

Atithe'tiine bf the invention, it would have been' obvious to a person of Ordinary skill in 
the art to uisel thelpirocess of modeling faces for face recognition, as taught by Nefian, with the 
process of face rpcognition and audio-visual speech recognition for identifying ^ subject* :as 
disclosed by Basu; • : 

TheisugigeStjpnymptivation for doing so would have been to provide an increased • 
recognition; dite CtSf^fian, Table 2). 

Therefore, :it would have been obvious to combine Basu with Nefian to obtain the 
invention as specified in claim 7. j ; 



As.to claim 8, the combination of Basu and.Nefian discloses the method of claim 7, 
fiirther coniprisihg proyiding the subject access to a restricted area after identifying the subject 
(Basu, column 11, lines 60-65). j 
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As to claim :10y the combination of Basu and Nefian discloses the method oi claim 7, 
further cpmprisjng obtaining observation vectors fromia sampling window of thie imagf; (Nefian, 

section 4)1 



As to claim jl 1 , the combination of Baisu and Nefian disclose the method of clairh i 0, 

.( 1 . ... 

wherein the obseryation vectors comprise discrete cosine transform coefficients] (iVe//an,: section 

. I :i ; - ■ ' • ■ 

41 



As to claini ;i2, the combination of Basu and Nefian disclose the method claim 7, wherein 
recognizing theifa-C? comprises performing a Viterbi decoding algorithm {Nefiairi, section 5). 



As to .cla!im 13, the combination of Baisu and Nefian discloses the method of claim 7, 
wherein recognizihg the audio-visual speech further comprises detecting and tracking a mbuth 



region. U3ing veptQri machine classifiers {Basu, column 10, lines 14-25), 



As:to claim 15, the combination of Basu and Nefian discloses the method of claim 7, 
further comprising results of recognizing the face and recognizing the audio-visual speech 
pattern iaceording ;tp a predetermined weighting to identify the subject {Basu, cdlumn 8, lines 43- 

■ ' V";' .i i ■ '* • ■ '* • ' - ' 

50). " 



: : I 
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6. ^l^ins 16; 17^ 20 and 21 are rejected under 35'iUiS.C. 103(a) as being unpatentable over 
Basu in vi^w of the article "A Bayesian Approach to Audio- Visual Speaker Identification" to 
Nefian et al. ("Nefian2;") (already of record). 

As to; claim 1 6, Basu discloses a system comprising: 

i ■ • 1 ? : • ' ' ' 

■ . ' , ■ (. ■ 

At least one capture device to capture audio-visual information from a subject (column < 
lines 15-23 and column 13, lines 31-67); 

A. first sjtoijage device coupled to the at least one capture device to store 6ode to (enable 
the systcmlcolumn 13, lines 31-67) to recognize a face of the subject from first lentries in a 
database (dhlUmn 6, lines 32-58, wherein the facial recognition of an individual\speaking is 
performed), recognize audio- visual speech of the subject from second entries in the database 
(column 5;; lin^s 2Br42\arid column 11, lines 10-31, wherein speech is recognized using the: audio 
of a audio-visual signal and that speech is verified using the utterance verifier, xyhich Uses the 
video of the audiprvisual signal)^ and identify the subject based on the face aiid |the audiprspeech 
(column 8, lines 43-50, wherein the recognition results determined by the face and audio-visual 
speech f^cpghizer)s are combined to produce a joint score which is used to ideriiify the 
individudl); ei^d y: : • 

A propesspr couplpd to the first storage to execute the code (column 13, lines 31^67), 

Basu does not disclose expressly, model an image including the face using an embedded 
hidden Maii^kpv'rnbdelj model the image and ari audio Sample using a coupled hidden Markov 
model, ahd;identify the subject based on the face and the. audio-speech according to. XfL(0^|k)+ 
XavL(0%0?^|k)].wh^?e Q*- and 0*^qre audio speech, visual speech and face of the paptiired 
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audio-visual ihfortriatipn, L(*|k) is an observation likelihood for a k^*^ entry in the databaise,; and 
A,fXav>=0^ Xf 4- >.a^=:l are weighting coefficients for face; arid audio-visual speech likelihood of 
recognition. [ , ■ ■ 

However, !Nfefian2 discloses to recognize a face of the subject from first entries in a 

'. . ? • ■ 

databiiseii model an iniage incliiding the face using an ^mbedded hidden MarkoV model (Sisction 
2\ model th^ image, and an audio sample using a coupled hidden Markov model {Section 3\ 
recognize audio-visual sjpeech of the subject from second entries in the database {Section 3\ and 

identify the subject based on the face and the audio-speech according to XfL(0^||c)+ 

•'':=■: s i • ' i • ' ■ i 

XavL(0\0r|k)4 whibfe 0^ 0^ and O are audio speech, visual speech and face of the captured 
audio-visM information, L(*|k) is an observation likelihood for a k^ entry in the database, and 
^^v>=0, ?ifH-;X^y=tl; are weighting coefficients for face and audio-visual speech likelihood of 
recognition {Section 5); and 

fiksii & 'Nefiaii2 are combinable because they aire from the same art of image processing, 
specifically recognition. 

At the time of the invention,, it would have been obvious to a person of ordinary skill in 
the art to use the process of modeling the face using an EHMM and the image and audio, siample 

I ;; ' ; i •; I ! • . • • ' • • • ■ ■ ' : ! 

using a CHMM, a.s:;Weil.jas, using the. identification prdcess, as taught by Nefiani2, with the : 

" i [•■\, ' " ' . : ' ' ' 

system disclosed by ;Basu. 

.... . . ^ 

The sugglestibn/motivation for doing so would have been to provide improved ^ ; 
identification accuracy (Nefian2, Abstract). - . 

Therefore,: it would have been obvious to combinie Basu with Nefian2 to' obtain the' 
invention as speciiiisd in claim 16. 
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AjSito clairii 17,1 the combination of Basu and Nefian2 discloses the system of claim 16, 

. I ii J i ■ i ' j • ' 'I ■ 

wherein the database is stored in the first storage divide (Basu, column 13, lines, 31-67 and 

column 3, lines 45-48). 

As to plairii ;20j please refer to the rejection of claim 1 6 above. . i I : I ! j 

' ' •":':''•:!•• ' , i ■■ . i . .* ■ 

As to claiiw.21, the article of claim 20- further comprising instructions that if executed 

enable the system;;to provide the subject access to a restricted area after the subject is identified 

{Basil, column ll^.tine^^ : : ''M? 

' ' ^ . ■ K . ■ .| ■ • *; ' 

• ' ■ *' : ! ; : ' ' " : ' * 

7. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over Basu and Nefian 
as applied to claim :7 above, and further in view of Nefian2. * i 

■ i^i; ■ i';M .1 ■ • ; ' ' ! * 

As to filaim 14, the combination of Basu and Nefian discloses the method of claim 7. 
The combination of Basu and Nefian does not disclose expressly wherein recognizing the 
audio-visual speech comprises modeling an image and* an audio sample using a coupled hidden 

i 1 • . ! . ' 

Markov tncidel.l ji ! \ ' \ j ! ' '! 

• ' 1 ;• i . ■ . . • . 

Nefidn2 discloses to recognize a face of the subject from first entries in a database, model 
an image including :the face using an embedded hidden Markoy model {Section 2), model the 
image and an audib' sample using a coupled hidden Mabrkov model {Section i), recognize audio- 
visual speech ioftlie subjept from second entries in the ;database {Section i), and identify the 

■>'.'. 

subject b;ased i>h::the face and the audio-speech {Section 5). 
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Basu, Nefiiah 8c Nefian2 are combinable because they are from the same; art of image 

: i \--M \ • • ■ ' ' ■ ' - !: 

processiirg, specifically recognition. ; 

At the time of the invention, it would have been obvious to a person of ordinary skill in 

? ' • , ^ 

the art to model an jmageiahd an audio sample using a coupled hidden Markov model, as taught 
by Nefian2, in theiaudio-visual speech recognition propess disclosed by the combinatioii of Bksu 
andNefiari; ■■':■■:■!'■ - ' • . -L " \ 

The: suggestion/motivation for doing sp would have been to provide improved 
recognition accirraqy (Afe^a«2, ^6^frac/^^ . . . 

Therefore, it would have been obvious to combiine Basu and Nefian with Nefian2 1;o . 
obtain the ihveritiori asi^ecified in claim 1^ i ^ ; 

' I ' , ' 

8. Claim'24- is 'rejected under 35 U^S;C. 1 03(a) as being unpatentable over Basu and Nefian 
as applied to claim 15 above, and further in view of Nefian2. ; j i : • 

• ! 'i ' : :■ i ■ ' . • • ■ • 

As to claim 24, the cqmbination of Basu and Nefian discloses the method of claim 15. 
The conlbiniationofBasu and Nefian does riot disclose express ; : : 

predetermined weighting corresponds to XfL(0^|k)+ A,aVL(0^0''|k), where 0^ O"; and Of;^e audio 
speech, visual sp)eech and face of the captt^ audip-visual information, L(*|k) is an observation 
likelihood for a k^ entry in the database, and XfXav>=0, + X^v=l are weighting coefficients for 
face and audio-vi^jjkl speech likelihood of recognition; ! 

; Hbwever, !N;efian2 discloses to recognize a face of the subject from first lentriesiih a, 
database j!mo]del .ah linage including the face using "an embedded hidden Markov model {Section 
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2), model the image and an audio sample using a coupled hidden Markov model (Section 3), 

', -I ' ■ 

recognize audio-visuali speech of the subject from second entries in the database (Section ij), and 

identify the subject based on the face and the audio^speech according to XfL(0 |k)+ 

A^vL(O^O^^k), where O^, and are audio speech, visual speech and face of the captured 

audio-visual information, L(*|k) is an observation likelihood for a k^ entry in the database!, ^d 

A,fA.av>=rO; + Xkv^li ar6 weighting coefficients for face and audio-visual speech ilikelihood^bf 

recognition (iS^c/j/on J); and 

Basu, Nefijan & Nefian2 are combijiable because they are from the same; art ;0f image 

' ■ ' ^' ' 

processing, specifically recognition. : ; 

At the: time of the invention, it would have been obvious to a person of dfdinary skill in 
the art to. ihciudd. the process of modeling the face using an EHMM and the image and audio 
sample using a C!riMM,:a^ well as, using the identification process, as taught by NefiaaZ^ with 
the system disclosed by the combination of Basu and Nefian. 

The: sugjgeStjon/mptivation for doing so would have, been to provide improved ; 
recognitiiba accuracy (A/e^ 

Therefore, it would have been obviou? to combine Basu and Nefian with Nefian2'tQ 
obtain the invention as, specified in claim 24. i : 

9. Claim 25 is rejected under 35 U:S.C. 103(a) as |being unpatentable over the combination 
of Basu ahd Nefian2 as applied to claim 20 aSove, and fiuther in view of Nefiaii. 

, As to clairii 25J the combination of Basu and Nefian2 discloses the articlie of claiin!20, 

■ i J , • .i . ; . ' : 

wherein the emheilded hidden Markov model (EHMM!) iis a hierarchical statistical model having 
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a parentlayeriponesponding to a super state of the EHMM and including a plurality of.iiodes to 
represent hidden liddes, each node referring to a plurality of second nodes of a child layer 
corresponding to. a ^tatb of the EHMM, the plurality of second nodes each referring to an ; 

, . '! I ■* , • • * t ' ' 

observation node \Nefian2, section 2). 

■ The cpmbiimtion of Basu and Nefi6n2: does^ not disclose expressly whereiin the stke Of thfe 
EHMM is described by a mixture of a plurality of Gaussian density , functions having diagonal 
covariancd matrices, i : ! . !. i 

However; .'Nefian discloses recognizing a face of a subject from first entries in a database 
(Abstract); includpg modeling an image including the face using an embedded hidden Markov 
model (EHMM), \yherein the EHMM is a hierarchical -statistical model having a parent layer 
corresponding ^o' a super state of tiie EHMM and including a plurality of nodes to represent 

■ ' ' i • ■ . ' ' . ■ * ' ■ . 

hidden nbdes, each node referring to, a plurality of second nodes of a child layer corresponding to 
a state of the EHMM, the plurality of second ttodes each referring to an observation node and 
wherein the state of the EHMM is described by a mixture of a plurality of Gaussian density 
functions haying diagonal covariance matrices (F/^. 2a, equation 1 and Section 3 ^ wherein 
square rioqes correspond to tfie parent node and the circle nodes correspond to. the child nodes 
or observation nodes and equation 1 corresponds to the mixture of Gaussian density functions 
having diagonal cbyariance matrices). • i . !:; j.^ 

Basu^ Nefiah2' &• Nefian are combinable because Ihey are from the same art of image 
processing, specifically recognition. ^ 

At the time the invention, it would have beeii obvious to a person of ordinary skill in 
the art;to. u§e the process of modeling faces for face recognition, as taught by Nefian, with! the 
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process ef face recognition and audio- visual speech recognition for identifying, a subject, as 
disclosed by the cbmbination of Basu and Nefian2. 

The; sugjgestion/mptivation for doing so would have, been to provide an ihfcreased • . * 
recognition r^te (Nefiahv Table 2). ' | . 

Therefore, iit would have been obvious to combine Basu and Nefian2 with Nefian.to : 
obtain the invention as; specified in claim 25. j 

» 

; Conclusion^ 
1 0. Apjplicanf s amendment necessitated the new ground(s) of rejection presented in this 
Office action.: Accordingly, THIS ACTION IS MADE FINAL. See MPEP § .706.07(a); 
Applicant is reminded :of the extension of time policy as set forth in 37 CFR 1 . l |36(a). ' \ • 

A shorten^ statutory period for reply to this final action is set to expire THREE 
MONTHS- from the mailing date of this action. In the event a first reply is . filed within J^/O 
MONTHS of the ihiailihg date of this final action and the advisory action is not mailed until after 
the end of:the;TH|lEE4MpNTH shortened statutory period, then the shortened $tatutory period 
will expire: oil the jdate the advisory action is mailed, aid any extension fee pursuant to:37 
CFR 1 .i36(a)rwill| bp calculated fi-om the mailing date of the advisory action. In no events 
however, will the statutory period for reply expire later than SIX MONTHS fi-om the datelof this 
final attibni ' 
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l;l: • Any inqiiiry concerning this communication or earlier communications from the 
examiner should be ^irec1;ed to Aaron W. Carter whose telephone number is (571) 272t7445. 

The examiner can ;riormally be reached on Sam - 4:30 am (Mon. - Fri.). 

. ( • ■ . • . . 

If attempts' to reach the examiner by telephone are unsuccessful, the examiner's ; 
supervisor, .rifiiv^ih Mehta can be reached on (571) 2'^2-7453. The fax phone number for the 
organization Avhere this application or proceeding is assigned is 571-273-8300. : ' • 

Information regarding the status of an applicatipn may be obtained fromithe Patent 
Application Iiiformati^on Retrieval (PAIR) system. .Stajtus information for published applications 
may be bbtaiiied 'from either Private PAIR or Public PAIR. Status mformation for unpublished 
applications is aVa(ilable through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access toithe Private PAIR 
system, cbntact the iEl^ctronic Business Center (EBC) .^t 866-217-9197 (toll-free). If you Would 
like assistance; from a USPTO Customer Service Representative or access to the automated 
informatioil systerii; call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 




